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Background / Context: 

Context for research interest in evaluating this program. 

Despite the central relevance of teacher quality for education system performance, the evidence 
base on “what works” to attract and motivate good teachers is weak. In both developing and 
developed countries, teacher pay is overwhelmingly based on educational attainment, training 
and experience, rather than performance. Yet variations in teacher performance, even within a 
single grade in the same school, are substantial (Rivkin et al. 2005). Hanushek and Rivkin 
(2010) has estimated that the “good teacher effect” on student learning outcomes is roughly 
equivalent to the effect of a 50% decrease in average class size in the US - a much costlier 
reform. Studies also indicate a weak correlation between teachers’ actual effectiveness and the 
most common proxies that school systems use in screening and promoting teachers, namely 
education and experience. 

In this context, a growing number of school systems - in both developed and developing 
countries - are adopting reforms to link teachers’ compensation more directly with their 
performance. Different performance measures are used (teacher attendance, principal and 
observer evaluations) but given the growing availability of student test data, the key metric in 
most new “pay for performance” schemes is measured improvement in student learning 
outcomes, either at the school level (in the case of a “group-based” school bonus) or classroom 
level (for individual teacher awards). 

Despite this growing experimentation, the evidence base is both limited and conflicting. 
Most of the evaluated experience with bonus or merit pay has been in the US, where both early 
studies (Cohen and Mumane 1986) and well-designed recent experimental trials in Nashville 
(Springer et al, 2010) and New York city (Springer and Winters, 2009) have failed to find 
positive impacts on student learning outcomes from the introduction of either school-based or 
individual teacher bonus pay. However, the most carefully evaluated pay-for-performance 
programs for teachers outside of the U.S., in Israel (Lavy 2009; Lavy 2002), rural Kenya 
(Glewwe, Ilias and Kremer, 2010), Andra Pradesh India (Muralidharan and Sundararaman, 
2009) have all found positive impacts on learning, with the India results particularly strong. 

Existing evidence leaves important policy questions unanswered. All the studies 
described above compare schools with and without pay for performance incentives. All were also 
experiments that we not sustained long enough for potential perverse behaviors (cheating, 
teaching to the test) to develop over time. We still have limited understanding of how teachers 
respond to differently structured incentive schemes (e.g. linear versus non-linear contracts), and 
how the ambitiousness of performance targets affects teachers’ and schools’ motivation and 
responses. Above all, none of the studies to date has provided very rich insight into exactly how 
performance incentives change teachers’ behavior in the classroom - where improvements in 
learning are produced. The theory of action behind a performance incentive is that it motivates 
the agent to adopt or intensify behaviors that contribute positively to the desired outcome. Thus, 
there is potentially much to be learned from evaluations that not only measure the final impacts 
of bonus pay, but also delineate the channels through which it operates. 
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Purpose/ Objective/ Research Question / Focus of Study: 

Description of the focus of the research. 

This study evaluates the impact of a group (school-based) teacher bonus program introduced in 
the Brazilian state of Pernambuco in 2008. Research questions are: i) Does setting school level 
targets and awarding bonus pay produce improvements in student learning? ii) Does setting 
school level targets and awarding bonus pay stimulate visible increases in teacher effort or 
changes in teaching practice that can explain improvement/lack of improvement in student 
learning? iii) Do school targets and bonus pay stimulate adverse behaviors? iv) Do more 
ambitious targets (ie, non-linear targets) stimulate higher performance? v) Does achieving the 
bonus stimulate greater or lesser school effort in the subsequent year? vi) Does the degree of 
trust and “social capital” in schools affect their performance? 

Outcomes measured in this study include student learning (on state-wide annual tests); 
graduation and repetition rates; teacher and student absence; teachers’ use of class time, 
materials and interactions with students; and social capital and trust in schools (measured on a 
globally validated questionnaire, applied to school directors). 

An innovation of this research is its use of a standardized classroom observation 
instrument (the Stallings classroom snapshot) to collect rich, classroom level data in a panel of 
1,200 randomly selected classrooms in a state-wide representative sample of 300 schools. The 
instrument captures changes in teachers’ use of class time, materials, interactions with students 
and students’ engagement with their work in response to the bonus regime. The instrument also 
permits more precise measurement of possible adverse effects such as diversion of time away 
from non-tested subjects and excessive focus on test preparation. 

Setting: 

Description of the research location 

Pernambuco is a relatively poor state in Brazil’s Northeast Region. About 60% of all primary 
and secondary (basic education) enrollments in the state are in 1,000 state-managed schools, 
which are the focus of the research here. Education outcomes are very low; Pernambuco ranked 
last among Brazil’s 26 states on the 2007 national education quality index and Brazil itself — 
although it has registered a strong improvement in PISA results over the past decade - still 
remains a middling performer within Latin America, with an overall PISA score below 400. 

Since 2007, however, a dynamic Governor has made education improvement a priority in 
Pernambuco and has adopted a number of impressive reforms, including an innovative school 
bonus scheme. 

Population / Participants/Subjects: 

Description of the participants in the study: who, how many, key features, or characteristics. 

The research is tracking both the overall performance of the state school system as well as 
detailed classroom-level changes in a panel of 300 (30% of the total) state schools. There are 1.3 
million students and about 50,000 teachers in the system. 

Intervention / Program / Practice: 
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The Pernambuco performance pay system is an annual bonus paid to all school employees based 
on how well they meet individual school level targets. The system sets targets to be attained by 
each school, for each subject (Math and Language) and grade (4th, 8th, 11th) tested annually on 
the SAEPE (Pernambuco State Learning Assessment). These targets for improvement in average 
test scores are combined with targets for improvements in student pass rates under a formula 
called the IDEPE (Pernambuco Index of Educational Development). This index has two 
components: test scores based on the SAEPE (Pernambuco state level learning assessment) and a 
measure of pass rates primary and secondary classes. Because the index is the product of both 
test scores and pass rates, it discourages schools from automatic promotion of children who are 
not learning, but it also discourages schools from excessively high retention rates, which is a 
phenomenon in Brazil, with average repetition rates in primary school in excess of 25%. 

Our identification strategy exploits three particular features of the bonus program design. 
First, school targets have been defined each year since 2008, following a set of decision rules 
that generated discontinuities in the specific targets faced by schools near the 25 th and 50 th 
percentiles of the performance distribution; just above these thresholds, schools face significantly 
more ambitious targets. Consequently, we are focusing the analysis on the impact these 
differentially ambitious targets have on otherwise similarly-performing schools. Second, the 
Government has set a performance threshold for the bonus: schools must attain at least 50% of 
their targets in order to receive any bonus. Between 50 and 100%, the size of the bonus is pro- 
rated in line with their achievement. Third, the State allocates a fixed pot of funding to the 
program each year (one month of the total state education payroll), which is distributed among 
the schools that attain at least 50 percent of their target. Thus, the individual bonus also depends 
on how well other schools perform. This design has resulted in 52% of schools receiving the 
bonus in 2009 (based on end-2008 student performance), with an average bonus of 1.8 months’ 
salary for all employees, and 79% of schools receiving the bonus in 2010, with an average award 
of 1.4 months’ salary. 

Research Design: 

Description of the research design. 

Our design has four components. First, we use a RD design to analyze the responses of 
similarly-performing schools to the non-linear (more ambitious/less ambitious) performance 
targets generated by the Pernambuco bonus design over successive years. Second, we exploit the 
State’s 50% threshold to analyze the responses of similarly performing schools that just barely 
achieve and just barely miss earning the bonus, over successive years of the program. Third, we 
conduct classroom observations in a large sample of schools using a standardized, internationally 
validated instrument to detect changes in teachers’ classroom practice linked to the 
ambitiousness of their schools’ targets, the fact of receiving or just missing the bonus, and 
general experience with a bonus regime over time. We also apply surveys to school directors to 
measure school-level social capital and trust, which we hypothesize could be an important factor 
in the teamwork needed for bonus achievement in fairly large schools (40+ teachers, 
1,000+students) under a system where bonus results depend on the outcomes for the relatively 
small minority of classrooms whose students are tested). Finally, we complement these 
strategies with a dif-in-dif analysis to try to assess the system-wide impact of Pernambuco state’s 
bonus program on student learning outcomes, vis a vis neighboring Northeast states (without 
bonus programs) and vis a vis municipal school systems in the state of Pernambuco. 
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Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 

We use administrative data on the State’s bonus system implementation (school “contracts” 
specifying their annual targets) and annual state student test and student flow data. We also use 
questionnaires annually applied to school directors to analyze the levels of trust and teamwork in 
the school. Finally, we are collecting annual data on classroom dynamics in over 1,200 
classrooms in a panel of 300 schools. 

Findings/ Results: 

Description of the main findings with specific details. 

Given long delays in release of the Pernambuco state test microdata as well as the microdata 
from the national learning assessment that we will use for the dif-in-dif analysis, our analysis of 
the overall bonus impact is still ongoing. (Both the 2009 and 2010 microdata are due for release 
in October 201 1). Data from the 2010 classroom observation round has also not yet been fully 
analyzed. Thus, the results reported here are preliminary: 

Schools with more ambitious targets achieved more progress. In almost every 
performance category (4th, 8th, or 11th grade; math or Portuguese), schools that fell on the 
“higher target” side of the performance cutoffs made larger test score gains than comparison 
schools just below the cutoffs. At the 25th percentile of performance, schools on the higher 
target side of the cutoff improved their average 8 th grade test scores in Portuguese by .31 SD 
more than schools just below the cutoff and in math, by .15 SD. At the second cutoff (just above 
and just below the 50th percentile in the performance distribution) and other tested grades (4th 
and 1 1th), impacts were in similar ranges but varied across subjects and, in a few cases, by cutoff 
point. Overall, at least over the very short term, higher targets in the presence of an attractive 
incentive in Pernambuco resulted in higher learning results by schools. 1 

Schools that just missed receiving the bonus in 2009 appear to have improved more than 
schools that barely achieved the bonus in 2009. A key research question is whether schools that 
received the bonus in 2009 (based on 2008 results) would be more motivated in the second round 
of the program or would exert less effort and coast. Controlling for 2008 test results and other 
characteristics, schools that barely missed the bonus for 2008 improved more than schools that 
barely achieved the bonus for 2008. It appears that not getting the bonus had a positive effect on 
schools’ motivation and performance. 

Schools whose teachers spent more time on instruction were much more likely to achieve 
the bonus. Classroom observations carried out in November 2009, just before the end-of-year 
student achievement exams, showed significant disparities in how efficiently teachers used 
classroom time for instruction. These differences — and differences in the share of time teachers 
were off-task or absent from the classroom — were highly correlated with a school’s likelihood of 
achieving the bonus for 2009 (paid in 2010). 

Conclusions: 

Description of conclusions, recommendations, and limitations based on findings. 



1 A different model used to instrument the targets for the discontinuities also showed evidence that, all else being 
equal, higher targets lead to higher school-level learning results over the short term. 
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In the coming months, we expect to complete a full evaluation of the impact of Pernambuco’s 
innovative school bonus program over its first three years of implementation. Our study is one of 
the longest-running and possibly most comprehensive studies of such a program in a developing 
country setting to date, given our efforts to analyze classroom-level teacher practice and the role 
of social capital in successful attainment of group (school-based) bonuses. The research will also 
shed light on the role of bonus design features - such as average bonus size and non-linearity of 
targets - in affecting the perceived strength of bonus incentives and schools’ responses. 
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